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Executive  Action 


Pending  guidance  from  the  "Board  of  Advisors",  here  are  specific 
action  items  and  guidance  issues  tor  OSS  '94  which  I  have  extracted  from  the 
attached  summary  of  OSS  '93  evaluation  results: 

Over-all  we  hit  the  mark,  but  three  major  changes  in  substance  are 
required  for  OSS  '94: 

a)  We  need  more  participation  by  top-level  U.S.  intelligence 
community  leaders,  and 

b)  We  need  to  shift  from  "descriptive"  to  "prescriptive",  with  halt-day 
tracks  that  cover  policy/funding,  collection,  analysis  tools  &  techniques,  and 
"hands  on"  displays  and  workshops. 

c)  We  need  to  shift  to  increased  focus  on  sources  and  users. 

It  is  clear  that  we  have  achieved  momentum  and  established  ourselves 
as  the  focal  point  for  leading  the  "reinvention  of  intelligence"  effort,  Still,  it  is 
also  clear  from  many  comments  (both  written  and  personal)  that  we  now  need 
to  begin  providing  "strategic  specifics"  that  can  be  discussed  and  digested  by 
attendees  as  a  group  and  in  smaller  working  groups.  We  need  to  provide 
participants  with  a  version  of  the  National  Performance  Review,  one  focused 
not  only  on  open  sources  of  intelligence  but  on  the  over-all  directions  of  the 
intelligence  community  as  a  whole  (to  the  extent  that  this  can  be  discussed  in 
an  unclassified  forum  with  foreigners  present). 

There  would  be  distinct  advantages  to  combining  our  2.5  day  session 
with  a  one  day  classified  session  hosted  by  the  Community  Management  Staff 
on  the  day  following.  Support  tor  this  idea  would  be  appreciated. 

We  will  invite  someone  from  the  existing  Presidential  and  Director  of 
Central  Intelligence  security  commissions  to  present  issues  and  directions 
pertaining  to  changes  in  security  classification  and  open  source  access. 

We  will  focus  firmly  on  U.S.  intelligence  community,  but  will  continue 
to  invite  selected  foreign  speakers  (e.g.  Director  of  International  Red  Cross)  to 
continue  to  explore  international  non-governmental  aspects  of  open  source 
intelligence  burden  sharing.  Our  foreign  speakers  will  be  pre-screened  to 


ensure  they  are  articulate  in  English. 

We  will  also  place  increased  emphasis  on  the  role  of  the  private  sector 
in  providing  intelligence  support  to  government,  and  on  "decision-support"  to 
the  American  Workplace.  This  will  be  done  with  guidance  and  assistance  from 
the  Department  of  Commerce. 

The  existing  format  of  the  conference  works,  except  that  we  will 
eliminate  evening  events  and  stop  at  1500  on  the  third  day.  We  will  also 
ensure  that  displays  and  interest  groups  are  available  during  the  morning  "shut- 
down" period  on  all  three  days  (this  year  the  Omni  could  not  give  us  the 
display  room  on  the  first  day). 

We  will  increase  displays  to  around  60,  with  special  emphasis  on 
adding  government  displays  and  government  representation  of  its  needs  for 
private  sector  support. 

We  will  increase  (if  government  supports)  amount  of  material  on 
government  open  source  requirements  and  capabilities  included  in  Volume  I 
(Background)  of  Proceedings. 

We  will  increase  the  number  of  interest  groups  in  order  to  keep  most 
groups  down  to  20  people;  and  will  add  several  100-person  audiences  for  five- 
person  "working  groups"  corresponding  to  each  of  the  "tracks". 

We  will  stay  in  the  Washington  D.C.  area  as  a  means  of  optimizing 
both  the  quality  of  our  speakers,  and  the  accessibility  of  the  symposium  to  our 
government  participants. 

We  will  continue  to  hold  the  symposium  in  mid-November  or  the  first 
week  in  December. 

We  will  not  cap  attendance,  although  we  are  limited  by  the  ability  of 
most  hotels  to  accommodate  our  preferred  banquet  seating  arrangements. 
Preliminary  survey  suggests  that  we  can  go  up  to  800  banquet  seats  and  200 
theater  seats  for  OSS  '94,  for  a  total  of  1,000. 

Our  weakest  area  was  clearly  audio-visual  (we  fired  the  old  audio-visual 
company  two  days  before  the  symposium  actually  began-although  our  new 
audio-vLsuai  was  much  better  than  the  old,  dramatic  improvement  is  required). 
There  are  three  things  we  can  do  to  improve  this: 


a)  While  retaining  the  "inaugural  type"  screens,  we  will  ensure  that 
viewgraphs  can  be  screened  properly,  and  will  also  instruct  our  audio-vidual 
technician  to  leave  the  slides  up  rather  than  fading  in  and  out  between  speaker 
and  slides. 

b)  We  will  provide  explicit  presentation  guidance  to  speakers.  Most  of 
our  speakers  were  not  expecting  a  "world-class"  room  with  an  audience  of  just 
over  800  at  peak  times. 

c)  We  will  evaluate  cost-benefit  of  pre-processing  speaker  slides  and 
viewgraphs  to  ensure  top-quality  graphics  readable  from  the  back  of  the  room. 

Registration  procedures  will  be  improved,  notably  by  keeping  all  badges 
together  in  alphabetical  order  rather  than  trying  to  set  aside  a  sub-set  tor 
speakers,  press,  and  guests. 

Laptops  ("tapping  noises")  will  be  isolated  on  one  corner  of  the  room. 
Side  conversations  in  the  plenary  session  will  be  discouraged.  A  special  room 
will  be  provided  for  off-line  questioning  of  speakers  immediately  after  their 
speech.  The  most  important  speakers  will  have  public  question  &  answer 
periods.  Smoking  will  not  be  permitted  in  the  ballroom  foyer  or  other 
conference  areas.  Badge  names  and  organizations  will  be  made  slightly  larger, 
and  "codes"  will  be  published  in  the  schedule  booklet. 

\  Pricing  structure  for  OSS  '94  will  be  along  lines  of  $750  corporate, 
$500  government,  and  $250  for  individuals  not  supported  by  an  organization. 
(NOTErMweliminary  accounting  indicates  that  we  will  be  able  to  cover  all 
expenses  awl  my  salary  at  the  GS-15  level,  but  that  we  did  not  achieve  the 
$50,000  buW-or  any  fraction  of  it-which  1  was  hoping  to  establish  as  a 
growth  fund.)V 
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Evaluation  Prepared  From  175  Completed  Booklets 


How  many  days  should  this  annual  symposium  last? 

1  day  =  0        2 days  =  33     3 DAYS=  116  4 days  =  8       NoAns  =  7 


How  long  should  a  typical  speech  last  In  minutes? 

20  min  =  39    30MDV=  101  40mln=  19    NoAns  =  3 

2  people  pointed  out  that,  "Good  speakers  with  good  stuff  should  be  given  time  to 
expand. 


Should  displays  be  manned  and  available  while  speeches  are  delivered? 
YES  =  122       No  =  44  NoAns  =  6 


Should  there  be  a  single  dedicated  display  day? 
Yes  =  22         NO  =  142        NoAns  =  6 

One  person's  comments:  '1  am  from  private  sector  -  private  sector  needs 
orientation  in  government  programs  that  government  people  don't  need.  This 
could  be  done  through  some  displays  by  some  agencies  specified  for  private 
sector  (don't  need  speakers  -  so  many  attendees  in  government  service  know 
this).  For  example,  I  had  never  heard  of  Tipster,  MUC,  TREC  until  yesterday  and 
I  got  a  good  crash  course  (finally)  on  the  3rd  or  4th  try  In  the  display  rooms,  but  I 
would  like  a  progress  report  on  government  projects  -  a  summary,  more 
deliberately  focused  for  private  sector  -  lists  of  keywords  so  I  can  search  the 
abstract  services  or  know  where  to  get  further  reading,  etc...." 


Do  you  like  "shutting  down"  the  Plenary  Session  for  two  hours  each  day? 
YES  =  122       No  =  37  No  Ans  =  1 1 


How  many  displays  should  there  be? 

30  displays  =  39        50  DISPLAYS  =  62     75  displays  =  40       No  Ans  =  15 

Nine  people  said  have  as  many  as  possible,  appropriate  and  for  which  we  have 
sufficient  space  with  low  noise  level. 


How  many  Interest  groups  should  there  be? 

10  GROUPS  =  92        20  groups  =  39  30  groups  =  8        No  Ans  =  13 

One  person  suggested  ,  "Room  for  larger  groups." 

Several  people  commented,  "Run  some  sessions  two  times." 

One  person  said,  "20  over  2-3  days  if  you  have  separate  display  day." 

Three  said  determine  the  number  by  as  many  appropriate  topics  as  possible. 
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What  size  groups  would  you  prefer  for  an  Interest  group? 

10  reOPLE  =  90        20  people  =  39  30  people  =  8        No  Ans  =  13 

One  person  suggested,  "the  number  of  people  depends  on  the  room  size  and  topic." 

Should  lunch  be  organized  and  included  in  price,  or  a  two-hour  free  period? 
ORGANIZED  =  137     Free  =  28        No  Ans  =13 
Two  people  said,  "Some  of  both" 

What  time  should  the  last  day  end? 
(Not  yet  tabulated) 

Given  the  cost  in  time  and  lost  substance,  would  you  still  prefer  frequent  regular  breaks, 
or  the  existing  major  break/leave  as  you  need  to  system? 

Frequent  =  35  OK  =  127        No  Ans  =  12 

Comments:     "Stress  this  policy  during  the  opening  remarks" 
"Fine  given  current  informal  seating  setup" 
"Fine  but  give  everyone  a  5  minute  stand  in  place  break  every  2 
hours  to  stretch" 

For  USG  Local  Only:  If  we  were  forced  to  move  out  of  town  every  second  year,  would  you 
budget  to  attend  or  send  someone? 

Yes  =  43         No  =  36  NO  ANS  =  95 

1  "No"  said  he/she  would  attend  if  the  symposium  is  held  in  Hawaii!!  (Hram...) 

US  Government  Only:  Please  list  the  month  in  which  you  would  prefer  to  attend  this 
event? 

Jan  =  2  Feb  =  1  Mar  =  3         Apr  =  2  May  =  1 

Jun=  1  Jul  =  0  Aug  =  0  Sep  =  2  Oct  =  7 

Nov  =  20         Dec  =  3  NOANS  =  113 

One  person  wrote,  "Same  time  each  year  to  establish  regular  attendance." 
Another  person  said,  "during  Congressional  break." 

All  others:  Should  this  event  be  held  in  Washington  every  year,  or  moved  around? 
DC  =  75  Move  =  29      No  Ans  =  70 

One  person  wrote,  "DC  because  the  best  speakers  can  be  persuaded  to  drop  in  for  a 
speech." 
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Another  person  suggested,  "DC  so  you  don't  lose  the  Intelligence  Community 
representatives  and  the  US  government  representatives." 
Another  person  pointed  out,  "If  you  move  you  may  be  able  to  attract  different 
types  of  people  but  be  aware  the  conference  will  have  a  different  flavor." 


All  others:  What  month  would  you  prefer  to  see  this  event  held? 

Apr  =  3  Jun  =  2  Aug=  1  Sep  =  9  Oct  =  14 

Nov  =  39         Dec  =  2  NO  ANS  =  89 

We  were  reminded  to  watchout  for  Holidays  and  other  conferences. 


If  we  had  to  choose  between  September  and  early  December  next  year,  which  would  you 
prefer? 

SEP  =  99        Dec  =  63         No  Ans  =  12 

Two  people  commented  that  September  Is  the  end  of  FY  for  US  government  not 
making  It  the  Ideal  month. 


In  addition  to  this  annual  event,  is  there  a  need  for  one  day  regional  seminars  with 
video  excerpts  from  the  Annual  Meeting  and  Special  Interest  Groups  meeting 
regional  needs  (Asia,  Europe,  Africa,  Latin  America,  West  Coast)? 

Yes  =  72  NO  =  77  No  Ans  =  22 

"No,  don't  water  this  one  down" 
'Yes,  but  no  video  tape,  100%  live" 

"Damn  good  idea.  Consider  San  Antonio  for  the  1st  regional  meeting"  - 
Evaluation  Book  #5  -  participant  left  his  name  and  numbers  with  an  offer  to 
help. 


If  this  event  were  held  in  San  Jose,  California  next  year,  would  you  attend  or  send  a 
substitute? 

YES  =  87        No  =  61  No  Ans  =  23 


Please  list  internal  symposium  aspects  or  external  environmental  aspects  you  would 
like  to  see  integrated  for  future  years  (shopping,  parking,  night  life,  METRO, 
anything  at  all  that  would  improve  the  experience  for  you). 

(Results  not  yet  tabulated) 


How  did  you  hear  of  this  symposium? 

WORD  OF  MOUTH  =  66  Direct  Mail  =  34        Jane's  =  11 

From  last  year  =  1 1  AU=  3  UIC=1 

Signal  =  1  OSS  Notices  =  1        Wired =  1 

E-Mail  =  1  Hack-Tic  Conference  =  1 
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No  Ans  =  9 

28  people  gave  multiple  answers,  most  Included  Direct  Mall  and  Word  of  Mouth 
and  a  few  Included  Internet. 


In  what  month  did  you  hear  of  the  event? 
Last  year's  conference  =  7 

Jan  =  5  Feb  =  1  Mar  =  0         Apr  =  13        May  =  8 

Jun=  11         Jul  =  5  Aug  =  24         Sep  =  2  OCT  =  39 

Nov  =  9  Dec  =  9  No  Ans  =11 


In  what  month  did  you  decide  to  attend? 

Jan  =  4  Feb  =  3  Mar  =  Apr  =  4  May  =  6 

Jun  =  9  Jul  =  8  Aug=  12         Sep  =  34         OCT  =  67 

Nov  =  7  Dec  =  4  No  Ans  =10 

4  said  "after  last  year's  conference" 

How  large  an  "Early  Bird"  discount  would  it  take  to  get  you  to  register  30  days  prior  to 
the  event? 

$50  =  23         $100  =  62       $150  =  39       $200  =  20       No  Ans  =  30 
"As  big  as  reasonable" 

Three  people  commented,  "its  immaterial  for  government  employees  there  are 
either  funds  or  no  funds." 


How  large  as  "Early  Bird"  discount  would  it  take  to  get  you  to  register  15  days  prior  to  the 
event? 

$50  =  57         $100  =  47       $150=15       $200=11        No  Ans  =  40 
Five  people  wrote  that  its  immaterial  for  government  employees  . 


If  we  decide  to  cap  attendance  at  750,  and  displays  at  50,  while  maintaining  our 

international  advertising  momentum,  how  many  weeks  in  advance  would  you 
start  worrying  about  getting  a  seat  or  a  booth? 

2  weeks  =  8     4  weeks  =  8     6  weeks  =  21    8  weeks  =  35    10  weeks  =  16 
12  WEEKS  =  38  No  Ans  =  27 

"Capping  the  attendance  defeats  the  purpose  of  OSS." 
"Don't  cap  attendance." 
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As  you  reflect  back  over  the  three  days  of  speakers,  please  list  your  top  five  speakers, 
with  your  most  rewarding  listed  first: 

The  following  people  were  listed  as  favored  speaker  (number  of  times  Is 
Indicated  next  to  their  name): 


TOFFLER  =  95 

Caldwell  =  37 
Treverton  =  27 
KGB  =18 
Steele  =  16 
Hlmelfarb  =  13 
Burwell  =  5 


Cerf  =  49 
Kahle  =  36 
Miller  =  27 
Schmidt  =  17 
Treverton  =15 
Ellas  =  12 
Shima  =  3 


Kapor  =  45 
Wallner  =  34 
Pale  =  26 
Baumard  =  16 
Politi  =  14 
McGlll=  11 


Pedtke  =  37 
Wood  =  31 
Hall  =  22 
Katsen  =  16 
Etheredge  =  13 
Leijonhielm  =  6 


Please  list  the  three  speakers  least  useful  to  you  personally.  In  terms  of  your 
expectations  (number  of  times  is  indicated  next  to  their  name): 


MALAUD  =  40 

Caldwell  =  16 
Etheredge  =  12 
Pale  =  10 
Wallner  =  8 
Treverton  =  5 
Pedtke  =  2 


Katsen  =17 
Schmidt  =15 
Hlmelfarb  =11 
Baumard  =  10 
McGill  =  7 
Miller  =  4 
Cerf  =2 


Shima  =  17 
Burwell  =  14 
Toffler  =  10 
Thompson  =  9 
Ellas  =  6 
KGB  =  4 
Steele  =  1 


Leijonhielm  =17 
Politi  =  12 
Wood  =  10 
Hall  =  9 
Kapor  =  5 
Kahle  =  4 


Comments  included: 

'Toffler's  -  too  broad  -  maybe  we  needed  more  time  with  them  to  get  more  to  the 
specifics." 

"Foreign  speakers  in  general." 

"Foreign  accents  are  difficult  late  in  the  day." 

"IC  officials  that  didn't  show  up." 

"I  cannot  honestly  categorize  any  speaker  In  this  symposium  least  useful' 
because  even  those  whose  subjects  were  of  lesser  interest  for  me  were 
informative." 


What  frequency  of  publication  would  you  preder,  and  how  many  pages  should  OSS 
Notices  be: 


Bi-Monthly :   5  pages  =  20  20 Pages  =21  30  pages  =  10 

40  pages  =  6  NO  AN8  =  118 

Several  people  noted  they  had  not  yet  seen  OSS  Notices 
Several  people  said  the  amount  of  pages  is  immaterial,  base  the 
length  on  good  content. 

Monthly:       5  pages  =14  10  pages  =  23  1        5  pages  =16 

20  pages  =  13  No  Ans  =  105 

Eleven  people  enjoy  the  present  length. 

One  person  suggested  publishing  quarterly  with  20  pages. 
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We  are  thinking  about  pricing  complete  access  to  our  INTERNET  set-up  at  $400  per  year, 
and  bulletin  board  (conferencing)  access  only  at  $200  per  year.  In  your  opinion, 
is  this: 

Too  Little  =  0  About  Right  =  46       Too  Much  =  57  NoAns  =  71 

1  person  suggested  a  connect  price  of  $20  per  month 


Comments  in  Evaluation  Booklets: 

#78  -    "Discussion  Internet,  etc.  have  to  be  directly  tied  to  OSCINT' 
"Thanks  again  for  all  your  work  and  insight." 


#82  -    "90%  of  the  world's  Info  is  In  printed  form,  e.g.  paper,  film  a  talk  about  this  Open 
Source,  its  adv.  and  disav  would  be  very  erilightening.  Also  machine 
translation." 

"  Have  interest  groups  meet  at  the  2  hour  break  -  not  after  5  p.m.  -  stop  at  5  p.m." 

#157  -  (can't  read  all)  "Interest  group  leaders  should  keep  to  subject  matter  at  least 
initially" 

#124  -  'You  need  better  video  services." 

#34  -    "Include  badge  colors/codes  in  a  handout,  not  just  on  the  bulletin  board." 

"Would  like  congressional  FACTS  on  desire  and  proven  record  on  support  or 
non-support  to  IS,  to  include  rationale.: 

"Also  would  like  someone  from  private  and  public  TV,  radio,  and/or 
newspapers." 

"Keep  smokers  far  away,  could  even  smell  smoke  at  my  table!" 

#74  -    "A  minor  point  -  you  need  someone  to  proofread  the  French  bios  -  numerous 
errors." 

"Need  to  re-emphasize  the  current  use  of  open  sources  in  all  intel  -  too  many  of 
your  speakers/participants  seam  to  think  that  government  intel  analysts  refuse 
to  use  unclassified.  That's  true  of  CIA,  but  not  DIA,  INR,  etc.,  as  you  know. 
They've  been  beating  a  strawman.  I'll  take  FBIS  over  NISA  any  day!" 

#23  -  "Be  aware  that  some  of  the  literature  available  in  the  ante  room  is  somewhat 
offensive,  e.g.  the  green  "Covert  Action"  flyer  and  the  "CIABase"  handout,  as  a 
minimum  you  may  want  to  voice  a  disclaimer  from  the  podium." 

#14-    "Well  run"  "Group  demos" 
"Fix  slide  problems" 

#80  -    "Outstanding  job  on  this  seminar  I'll  be  back!  Semper  Fi!" 

#  122  -  "Not  enough  info  about  services  to  evaluate." 

"Don't  try  to  provide  something  that  already  exists." 

"Subscriber  fees  are  a  turn-off;  e.g.  youVe  already  recouped  costs  of  Proceedings 
and  OSS  Notices  or  would  this  lead  to  lower  symposium  costs?" 
"The  audio/visual  concept  was  great;  its  execution  was  terrible." 
"Never  hold  it  at  Omni  again." 

"Need  to  balance  METRO  access  (good)  with  driving/parking  access  (not  good  at 
Omni)" 
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"Amenlties  (coffee,  etc.)  should  be  continuous  and  not  frequently  run  out." 
"Interest  groups  should  not  run  concurrently  w/  plenary  sessions;  have  fewer 
with  options  for  interest  groups  or  displays.  If  possible,  have  multiple  sessions 
of  interest  groups  (say  2)  so  that  people  can  attend  more  than  one.  Also,  provide 
time  (brief)  at  plenary  session  for  "reporting  out"  from  interest  group 
moderators." 

"  After  2  iterations,  we  have  reached  the  point  where  the  symposium  should  have 
clearly  defined  objectives  and  speakers/interest  groups  designed  to  achieve 
those  objectives  (e.g.  one  objective  might  be  to  inform  on  key  issues).  You  might 
want  to  circulate  draft  objectives  to  selected  Individuals  for  comment/feedback. 
From  the  government  perspective,  the  symposium  needs  to  provide  information 
(factual,  quantitative)  that  supports  planning  and  programming,  the 
symposium  should  encompass  Intellectual  arguments,  but  they  don't  sell 
programs!" 

"With  respect  to  foreign  participation,  recognize  that  there  are  significant 
differences  in  the  structure  of  foreign  governments  and  their  relationship  with 
their  private  sector  which  causes  their  experience  not  to  [. . .  J  well  to  USG  ways  of 
doing  business  and  their  way  is  not  an  option  for  us." 

#70  -    "While  some  sophisticated  audio/visual  equipment  was  available,  it  didn't  work 
reliably.  I  would  prefer  less  audio  visual  sophistication  and  go  for  reliability. 
Also,  you  need  some  guidelines  for  speakers  for  preparation  of  slides.  Many 
slides  were  unreadable  for  the  size  they  were  magnified  and  distance  they  were 
viewed  from." 

"Navigating  the  INTERNET  session  was  mlsadvertlsed:  this  was  on  development 
of  INTERNET,  not  finding  you  way  around.  I  was  disappointed!  Also  too  small  a 
space  for  this  meeting." 

"The  names  of  people's  employers  is  too  small  on  the  badge.  I  can't  read  it,  even 
within  10  feet,  and  yes,  I  do  have  my  glasses  on.  Part  of  the  fun  of  this  Is  seeing 
who  is  represented  at  a  conference  like  this.  Make  the  print  bigger!" 
"Hard  copies  of  presentations  would  be  useful!" 
"INTERNET  access  at  the  conference  would  be  great!" 

'Your  sensitivity  to  the  anonymity  of  your  participants  is  greatly  appreciated." 
'Too  small  a  space  for  the  CIA  open  source  talk  (Kennedy  room)" 
"Ban  the  use  of  laptops  -  it  is  very  distracting  to  set  next  to  someone  who  is 
making  all  that  noise!" 

"Steele  really  kept  everything  on  schedule  and  had  great  comments  to  keep  us  on 
course  and  maintain  continuity  and  relevance.  Really  great  job!" 

#90  -    "As  a  Collection  manager,  I'd  like  to  better  know  what  specific  types  of 

information  are  available  that  could  be  of  use  to  my  analysts.  The  philosophical 
discussion  are  wonderful,  keep  them  coming,  but  focus  more  on  the  users:  What's 
Where,  How  you  can  get  it,  How  other  people  have  used  it.  Policy  makers  who 
make  the  call  of  how  extensively  open  sources  are  used  with  their  purview  need 
to  have  tangible  examples,  i.e.  "Success  Stories:"  specifics  are  paramount." 
"  Regarding  validation  issues:  when  used  In  an  Intelligence  environment, 
validation  is  up  to  the  analyst.  In  an  all-source,  integrated  intelligence 
environment  it  is  a  Part  of  the  big  picture,  a  component  of  the  end  product.  Even 
technical  intelligence  disciplines  are  subject  to  conjecture,  the  analyst  is 
empowered  to  decide  what  information  is  accurate.  Don't  sweat  validation. 
"The  international  flavor  was  great!  Keep  it  going." 
"Good  set-up.  the  big  screens  were  very  helpful.  Would  it  be  possible  to  have 
blacklines  of  the  slides  prior  to  presentations?" 
"Kudos  for  keeping  on  schedule.  It  makes  a  difference!  Good  job  again." 
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"Next  year:  Let's  discuss  -  more  responsibility  -  security/ classification  (we  won't 
consider  changing  the  current  system  until  someone  starts  pushing/questioning 
for  it).  Let's  see  more  user  success  stories.  We  also  need  to  offer  possibilities  to 
users  on  how  to  configure  their  open  sources  work  station.  What  set-up  will 
allow  best  access  to  databases?  Let's  get  less  philosophical  and  more  practical  - 
Who's  using  it?  What's  working?  What  are  some  problems?  We're  already  sold  on 
the  product.  How  do  we  get  it?" 

#62  -    "Give  breaks  more  frequently  and  shorter." 

"Supply  attendees  with  advance  copies  of  speaker's  slides." 
"Allow  time  for  Q&A  for  each  speaker." 

"Get  people  to  talk  to  the  group  rather  than  reading  a  speech.  No  matter  how  good 
they  are,  it's  boring  to  be  read  to." 

"Suggest  that  if  people  want  to  have  private  conversations  while  speakers  are 

talking,  they  should  leave  the  room." 

"You  did  admirably  well  in  keeping  to  the  schedule." 

"Food  service  was  excellent!" 

#18  -    "Excellent  symposium" 

"Impressive  hi-level  of  speakers." 

"Be  sure  to  capture  better  speakers  in  newsletter  articles!" 

#39  -   Q&A  time  for  each  speaker  at  the  podium." 

"Preview  graphics  before  use  to  see  how  good/bad  they're  going  to  look.  Some 
were  unreadable." 

#71  -    "Speakers  must  be  advised  to  use  large  contrasting  lettering  on  slides.  Lunches 
were  good,  Timeliness  was  appreciated!  Seems  to  be  a  disconnect  between 
speakers  and  their  slides.  Perhaps  a  small  TV  monitor  on  speaker's  podium  will 
help." 

"Though  logistically  a  nuisance,  handouts  of  speakers'  slides  would  be 
valuable!!" 

"OS  info  on  encryption  systems." 

"Steele's  slides  would  have  been  useful  handouts." 

"Switching  back  and  forth  between  projected  images  of  speakers  slide,  to 

speaker,  back  to  same  slide,  etc.  is  annoying.  Suggest  leaving  slide  on  screen 

until  speaker  is  finished  with  it,  usually  with  remark  "next  slide  please."" 

"Projection  system  is  pretty  good." 

"Power  lined"  tables  for  laptops  a  good  move." 

#7  -     "This  is  not  an  evaluation  form  —  it  is  a  survey  seeking  input  on  Vhat  is'  without 
eliciting  reactions  to  "what  should  be',  the  chief  problem  with  this  conference  is 
that  it  offers  no  framework  for  the  IC  to  understand  the  problems  related  to  OS 
usage  and  delivery,  state  of  the  art  examples  of  OS  programs  (drawing  on  work  in 
academia)  and  an  opportunity  to  debate  strategies  for  the  next  few  years.  A 
conference  is  needed  that  does:  (1)  What  is  the  status  of  what  we  know  about 
OS/digital  library  development  (2)  What  is  the  key  R&D  that  is  underway  -  where 
are  the  gaps  (3)  What  are  we  learning  about  user  behaviors,  etc.  from  operational 
testbeds  in  IC  and  academia  (4)  a  closing  panel  that  includes  IC,  industry, 
academia  reps  on  Where  do  we  go  from  here." 

"Smoke  Free  environment  please.  This  is  the  only  conference  I  have  attended  in 
past  5  years  where  smoking  is  permitted  in  hotel  hallways." 


#30  -    "What's  new  with  respect  to  CAS,  Dialog,  BRS,  etc.  services" 
"Need  a  brief  description  of  all  the  displays." 
"Eliminate  prototype  projects." 

#108  -  "like  hi-tech  slide  capability,  but  need  to  practice  more.  Keep  slides  on  screen, 
not  person." 

#150  -  "It  was  great  to  have  been  here!  Thank  you  and  your  staff!" 

#87  -    "Robert,  I  came  to  this  symposium  because  I  personalty  believe  In  it,  and  what 

you  are  trying  to  achieve...  I  find  it  difficult  to  answer  some  of  the  questions  with 
respect  to  costs  $$$... this  Is  because  I  covered  a!  costs  myself... My  university  will 
only  normally  support  us  to  a  value  of  $1000-$1500  (Australian)  and  I  have 
already  "used"  that  amount  up!  (poor  university!)  I  have  cost-justified  this  trip  by 
bringing  my  wife  and  believing  it  to  be  a  holiday!!!  P.S.  I  consider  that  this 
symposium  is  very  valid  to  me  and  if  I  can  afford  it,  I  will  return  next  year.  I 
intend  to  make  it  my  business  to  "report"  (advertise)  the  proceedings  back  to 
Canberra  and  Australia  in  general." 

"It  was  a  pleasure  to  give  an  abbreviated  talk.  I  was  disappointed  with  one  of  my 
colleagues:  Jean-Marie  Bonthous  spoke  after  me-taking  40  minutes,  instead  of 
15,  "walking  over"  the  time  left  for  one  of  his  countrymen  Phillipe  Baumard!  (7 
minutes  left).  Never-the-less  It  was  well  received.  I  focused  on  cultural- 
intelligence  In  the  time  sense  of  the  word.  Jean-Marie  spoke  much  more  about 
business-intelligence  in  Germany,  France,  Sweden  &  USA.  Phillipe  spoke  about 
models  of  economic-intelligence  and  Japanese  and  German  banks  compared  to 
French  banks  (this  was  the  same  presentation  as  his  presentation  In  the  plenary 
session  on  Thurs!)" 

Re  94:  "Cultural  intelligence,  Hackers  "Cientists  of  Cyberspace",  The  costs  of 
information  -  how  much?,  Intelligence  and  security  education.  Searching  and 
evaluation  of  open  sources,  Methodologies,  Case  studies/histories,  New  uses  for 
old  information." 

#  173  -  "I  would  think  you  might  want  to  consider  individual  vs.  commercial  prices.  I 
consider  pricing  commercially  acceptable  -  but  personally/lndivldualty  too 
high." 

Re  OSS  '94:  "Possibly  detail  actual  case  studies  as  with  Holden-Rhodes  last  ear.  I 
like  Baumard." 

#121  -  "Robert  -  thanks  again  for  a  great  symposium,  {personal  remarks]  Here's  hoping 
you  and  OSS  out  live  all  of  them." 

#20  -    "Must  improve  the  vu-graph  system!!" 

#43  -    "Steele  and  company  did  a  super  job  well  thought  out  and  very  meaningful!" 

#153  -  "Dont  be  such  a  stickler  for  time.  Some  speaker  were  pushed  off  the  stage  without 
getting  their  full  30  min.  Some  time  for  Q&A  needs  to  be  worked  into  the 
schedule  next  year.  Don't  have  speakers  use  translators.  Takes  too  much  time 
and  is  hard  to  follow.  Just  have  the  speech  read  if  need  be." 

#131  -  "List  of  attendees  and  their  affiliations  would  be  useful  -  even  if  only  those  who 
pre-reglstered.  Author  index  to  proceedings  would  be  helpful.  Also  would  be 
helpful  to  have  a  list  of  the  displayers/exhibitors." 
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#100  -  "Don't  use  red  on  VG's  did  not  come  through.  Pass  out  copies  of  VGs  before  hand." 

#84-    "Did  not  get  a  clear  indication  of  purpose,  goals  of  the  symposium." 

'Too  many  speakers  simply  read  prepared  texts.  Not  an  engaging  speaker." 
"Visual  aids  need  overhaul." 

#95  -    "Have  the  slides  on  longer  so  that  they  can  be  read." 

"More  group  participation  in  concurrent  smaller  groups" 

#160  -  "Audio  visual  was  poor." 

"Arthur  Miller  session  too  long  became  chaos." 

#27  -    "Please  schedule  more  hands  on  applications  of  OSCINT  for  special  interest 
groups  meeting  &  discussions." 

#174  - "  The  slides  are  hardly  visible  to  someone  sitting  at  the  back  of  the  hall." 

"Speakers  should  distribute  their  material  beforehand  and  only  elaborate  on 
them  to  allow  more  time  for  questions." 

"A  list  of  participants  and  where  they  stay  would  be  desirable  for  private 
consultations  during  the  symposium." 

#175  -  "Interest  group  meetings  were  most  helpful  because  people  could  exchange  ideas 
on  specific  topics.  I'd  like  more  of  these.  Sometimes  the  plenary  session  speeches 
were  too  broad  and  general  to  be  realty  useful.  Also,  a  greater  focus  on  business 
and  the  private  sector  would  be  good  for  my  organization." 

#10  -    "Audio  visuals  were  extremely  poor.  Need  improved  slides,  improved  projection 
system  and  keep  slides  up  while  speaker  is  talking." 

#159  -  "Conference  displays  not  close  enough  to  main  ballroom  -  walk  too  long!" 
"Book  display  in  Senate  room  never  materialized  on  Thursday." 
"I  think  it's  great  that  you  ask  for  feedback  -  you're  sincere  about  wanting  to 
present  a  quality  conference  that  will  be  of  value  to  everyone." 
"I'm  sure  Keiji  Shima  is  quite  prominent  in  his  field  but  the  format  of  his 
presentation  was  hard  to  follow  and  didn't  address  the  subject  of  information  to 
any  great  degree  -  I'm  sure  he  has  plenty  of  info  to  share  but  he  didn't  share  it 
with  us.  Too  many  generalities  and  obvious  points  and  not  enough  specifics 
about  what's  going  on  in  Japan  re  open  source  information.  His  translator  was 
good;  she  had  a  very  hard  job." 

#12  -    "Please  find  technical  people  capable  of  running  your  multimedia  slides/video. 
In  this  day  and  age,  it  is  inexcusable  and  unprofessional  to  be  typing  in  slides 
during  a  presentation.  The  slides  were  illegible,  had  typos  and  were  slow. 
Technically',  this  conference  was  a  complete  failure." 

"The  conference  registration  appeared  to  be  unorganized.  Please  consider  getting 
a  conference  organization  to  help  with  these  issues." 
'You  may  consider  breaking  up  the  conference  into  political/technical 
solutions.  Maybe  dedicate  a  day  to  each." 

#88  -    "Improve  graphic  support  -  better  quality  graphics  -  handouts  of 
graphics/ abstracts  of  each  presentation." 

#1  -     "Great  conference." 
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#155  -  "More  panels  (perhaps  one  a  day?),  Better  technical  aids  (problems  with  video 
screens  became  ludicrous  and  were  a  constant  distraction,  8:00  is  too  early  to 
start  try  8:30  or  even  better  9:00." 

#50  -    "  Better  Video.  Pass  out  copies  of  slides.  Too  many  speakers  in  one  day.  More 
short  breaks  -  too  difficult  to  sit  for  3  hours  -  onfy  choice  is  to  leave  and  miss 
something." 

#13  -    'Well  done  will  continue  to  participate." 

#149  -  "Plagued  with  technical  and  organizational  problems  -  especially  disastrous  for 
a  gathering  this  large." 

#32  -  "Lighter  food  for  the  noon  meal.  It  was  just  too  much  and  too  heavy,  but  good!" 
"Robert,  you  did  a  great  job  in  keeping  the  speaker  to  the  schedule  -  that  is  very 
important  to  the  people  attending  the  event." 

#73  -    "As  I  sit  in  the  final  panel  (ex-KGB)  my  mind  goes  to  a  photograph  I  saw  taken 
recently  in  the  KGB  headquarters  courtyard  in  or  near  Drogobych  Ukraine.  The 
photograph  is  a  row  of  skulls  dug  up  in  the  courtyard.  The  skulls  belonged  to 
human  beings  who  were  killed  by  the  KGB.  I  was  looking  forward  with  interest  to 
what  the  panel  had  to  say,  but  suddenly  find  their  presence  here,  distasteful,  to 
say  the  least.  I  do  not  believe  that  it  is  a  credit  to  anyone  that  the  professional 
experiences  of  these  men  are  found  to  be  so  interesting  that  the  atrocities 
committed  by  their  organization  are  deemed  inconsequential.  Otherwise, 
excellent  job  at  pulling  this  off  again!  Mr.  Steele  and  staff  are  to  be 
congratulated!" 

#171  - "  Focus  seminar  on  Intel  Community,  its  use  of  open  sources,  it  problems,  where 
is  it  going,  what  are  its  requirements,  who  are  the  major  "real"  players  who  can 
be  contacted  for  inter[...]-  not  agency  heads,  coordinators." 
"I  do  not  listen  to  speakers  who  read  speeches.  Most  speakers  were  extremely 
boring  because  of  this.  Need  better  admin  operation  of  conference  - 1  received  no 
agenda  or  schedule  prior  to  arriving;  I  was  not  notified  that  I  was  to  be  a  panelist 
until  I  arrived;  audio/visual  support  was  terrible,  moderators'  comments  (Mr. 
Steele's)  we[. . .  J  often  erroneous,  not  founded  in  fact,  and  represent  opinion  not 
fact.  He  should  exercise  some  professional  discipline  in  this  area." 

#3  -     "I  was  quite  disappointed  in  the  symposium.  It  didn't  work  well,  Nix  the  foreign 
speaker.  Lets  focus  on  what  need  to  be  done  to  improve  US  Intel  using  OSCINT; 
identify  problems  and  develop  solutions.  To  do  this  we  need  to  get  intel 
collectors,  processors,  analysts  and  users  to  speak  openly  and  frankly  (How?  I 
don't  know)  and  then  convene  working  groups  (including  reps  of  above  groups)  to 
devise  possible  approaches  for  addressing  current  shortfalls  and  anticipating 
future  needs.  I'm  not  sure  what  needs  to  be  done  to  achieve  this,  but  that's  why 
they  pay  you  the  big  bucks!!" 

"A  format  and  collection  of  speaker  that  work  toward  identification  of  systemic 
problems  and  develop  possible  solutions.  I  think  most  of  us  recognize  that  the  IC 
is  confused  about  their  role  in  supporting  national  security  and  competitiveness 
and  has  little,  if  any,  motivation  to  move  intelligence  production  into  the 
1990's.  True,  they  don't  have  a  clue  how  to  utilize  information  effectively  in  this 
day  and  age,  but  I  think  the  former  problems  are  more  pressing.  Ideally,  this 
symposium  would  address  all  these  problems:  (1)  stimulate  a  discussion  of  how 
IC  resources  are  best  used  in  support  of  policymakers  (value  added  over  CNN  but 
timely  and  purposeful  ~  aimed  at  supporting  defined  National  goals  and 
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strategles,  (2)  updating  the  Ingrained  IC  culture,  (3)  Identifying  bottle  necks  In 
OSCEMT  Intel  collection,  processing  and  analysis  and  (4)  development  of  a 
system  to  address  these  problems.  In  my  opinion,  the  primary  OSCINT  need  of 
the  IC  is  a  method  for  filtering  and  sorting  relevant  info  and  delivering  a 
manageable  subset  of  the  "best"  Info  for  the  analyst  to  digest.  Addressing  this 
need  Is  a  multlfaceted  problem  Involving  changes  In  hardware,  software  and 
culture.  The  IC  needs  to  de  dragged  (they  will  be  kicking  and  screaming)  Into  the 
Information  age  of  the  1990's  and  beyond.  I  think  the  OSS  Symposium  can  help 
enormously  In  this  regard,  but  I  don't  think  It  got  very  far  this  year.  A  better- 
defined  goal,  clearer  focus  and  a  more  Interactive  format  (workshops  for  group 
discussion,  not  "networking")  will  help.  I  learned  a  lot  about  info  acquisition 
(this  was  great)  but  a  specific  connection  to  the  IC  was  lacking." 

#106  -  "Post  name  of  SIG  outside  door,  and  post  room  changes  in  same  place." 

#72  -    "Why  have  a  French  man  with  bad  English  read  another  Frenchman's  speech? 
Why  blame  SCI  for  your  ineptitude  in  determining  your  orgs  Intel  needs  and 
appropriate  sources?" 

#35  -   "Don't  not  give  equal  time  to  all  speaker.  Vary  time  according  to  material  to  be 
presented  -  maximum  one  hour." 

#58  -    'You  really  need  to  get  someone  to  help  you  with  your  organization,  and 
meeting-related  pubs  -  might  save  you  from  things  like  Wednesday's 
embarrassing  (I  expect)  2  hour  lapses  (Thursday's  too!)  -  typos  in  your  bio's." 
"The  hotel  was  great  (except  doors  at  back  of  room  made  lots  of  noise!  and  went 
all  the  time.)  The  food  and  food  amenities  (rolls,  fruit,  coffee  service)  were  as 
good  as  could  be  asked  for. 

"A  map  showing  all  rooms  (Interest  groups)  in  program  would  help." 
"Really  enjoyed  extraordinarily  high  standard  of  speakers" 
"Schedule  brutal  -  exhausted  by  end  of  3rd  day." 

"Remind  your  speakers  (if  you  feel  you  can  do  so  without  offense)  that  they  speak 
to  a  highly  intelligent,  highly  educated  audience  (repetition  &  redundancy)  while 
occasionally  rhetorically  satisfying.  Is  not  necessary  to  make  a  point  to  this 
audience  (I  had  a  prof  in  grad  school  who  beat  me  to  death  (almost)  on  this  point 
0  I  haven't  forgotten!) 

#15  -   'Your  'proceedings'  appear  111  organized  and  unrelated  to  actual  presentations. 
Clean  up  presentations  copies  and  include  a  preface  to  explain  flow  of  the 
materials." 

"The  tendency  of  Bob  Steele  to  identify  'his'  conference  with  himself,  is  jarring. 
Reduce  your  profile  and  broaden  your  credibility!" 

#66  -    "I  feel  less  speakers  are  necessary.  Most  repeat  same  Info:  'information 

technology  Is  vital'.  We  already  know  this.  More  displays  and  opportunities  to 
meet  colleagues  who  work  in  same  areas  Is  where  the  conference  is  important." 
"More  general  advice  by  speaker  rather  than  pressure  sales  by  speakers  for  their 
products.  Decision  needs  to  be  made  as  to  whether  this  is  going  to  be  an 
informational,  academic  conference  or  a  sales  conference,  In  which  it  should  be 
substantially  cheaper." 

#158  - "  Keep  the  personal  computer  out  of  conference  room  -  tapping  while  speakers  are 
presenting  is  rude.  Remind  attendees  private  discussions  are  appropriate  only  in 
the  foyer  -  not  in  conference  room  during  presentations,  question  and  answer 
periods." 
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#133  -  For  OSS  94  "Copies  of  view  graphs  and  slides!!!  METRO"  Cokes  in  morning  (pre 
0800)" 

#  162  - "  I  would  like  to  hear  a  case  study  or  two  of  how  open  source  information  was 

successfully  applied  to  a  government  or  business  intelligence  problem.  I  would 
also  like  to  see  more  representation  of  business  intelligence  activities." 

#19  -    "Superb  array  of  excellent  speakers.  Very  interesting  and  informative." 
"Audience  should  sit  up  closer  to  speaker." 
"Post  attendee  list." 

"Provide  attendees  with  list  of  attendees." 

#165  -  "More  people  from  private  sector" 

"Can  military  lingo  '  table  commanders'..." 

"Off-line  room  for  follow-up  with  questions  is  a  good  idea." 

#98  -    "  Runs  too  late  on  dark  evenings  when  local  people  have  a  long  trip  home  and 
family  responsibilities.  Handling  slides  was  dreadful  this  session.  Please 
provide  copies  of  slides  and  papers  ahead  of  time  so  we  can  pay  attention  to  the 
speakers.  Have  vendor  hours  between  8  &  9  AM." 

#94  - "  Themes  by  day  or  half  day."  Plus  some  comments  I  couldn't  decipher. 

#116-"  Please  make  sure  that  foreign  speakers  do  have  well  focused  presentations 
instead  of  anecdotes  of  un-related  speech  (e.g.  Philippe  Malaud)" 

#134  -  "The  informality  of  seating  etc.  is  a  big  plus." 
For  94  'Interest  groups  to  stay  on  topic." 

#154-  "Regional  symposium  for  distant  (market)  areas." 
"Tie  in  with  SCIP  chapters  throughout  country." 

"Stress  collaboration,  dependency  as  the  basis  for  security  in  the  new  world. 
Merger  and  acquisitions  rather  than  competition  (Himelfarb  of  SCIP  is  a 
specialist  in  mergers).  Excellence  as  the  basis  for  collaboration.  Excellence  seen 
best  in  open  sources  -  in  all  their  color  a  variety  and  freedom  in  form  (not 
pressed  into  a  mold  as  done  with  the  other  INTs).  Variety  poses  challenge  to  info 
managers:  collection,  processing,  storage,  retrieval,  use.  Variety  challenges  info 
technology  to  deal  with  multi-media,  esp.  color  TV  real  time,  translated 
simultaneously  retrievable  for  use.  Plethora  of  information  challenges 
imaginative  innovation  of  means  to  access  in  a  user  friendly,  user  seductive. 
user  cuddly  way.  Plethora  of  (...]  holders  poses  challenge  to  redefine  the 
paradigms  of  the  {...}:  author-creator,  publishers,  providers,  value-adders, 
intermediary  service  points,  end  users/customers,  therefor  laws  need  to  be 
changed  in  context  of  new  international  conventions.  New  challenges  in 
multinational  equities,  multinational  corporations  -  International  info 
entertainment  industries,  Banking,  Internet,  Decentralization  &  leveling." 
Name  and  phone  number  included. 

******************** 

The  above  relates  comments  from  booklets  completed  and  returned  at  the  conference. 
We  are  still  recieving  booklets  in  the  mail,  also  other  separate  written  comments  from 
the  conference  have  not  yet  been  included.  A  summary  will  be  included  in  OSS 
NOTICES,  and  complete  report  in  Volume  n  of  Proceedings. 


